<!DOCTYPE html>
<html lang="en">
<head><meta name="generator" content="Hexo 3.8.0">
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="google-site-verification" content="xBT4GhYoi5qRD5tr338pgPM5OWHHIDR6mNg1a3euekI">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <meta name="description" content="威龙的博客">
    <meta name="keyword" content="随缘">
    <link rel="shortcut icon" href="/img/favicon.ico">

    <title>
        
        TB级NFS数据平滑迁移方案设计与实现 - 威龙的博客 | VeyronC&#39;s Blog
        
    </title>

    <!-- Custom CSS -->
    <link rel="stylesheet" href="/css/aircloud.css">
    <link rel="stylesheet" href="/css/gitment.css">
    <!--<link rel="stylesheet" href="https://imsun.github.io/gitment/style/default.css">-->
    <link href="//at.alicdn.com/t/font_620856_pl6z7sid89qkt9.css" rel="stylesheet" type="text/css">
    <!-- ga & ba script hoook -->
    <script></script>
</head>

<body>

<div class="site-nav-toggle" id="site-nav-toggle">
    <button>
        <span class="btn-bar"></span>
        <span class="btn-bar"></span>
        <span class="btn-bar"></span>
    </button>
</div>

<div class="index-about">
    <i> 心若水， 自非凡。该博客已迁移至 https://www.cnblogs.com/xrszff </i>
</div>

<div class="index-container">
    
    <div class="index-left">
        
<div class="nav" id="nav">
    <div class="avatar-name">
        <div class="avatar">
            <img src="/img/avatar.jpg">
        </div>
        <div class="name">
            <i>威龙</i>
        </div>
    </div>
    <div class="contents" id="nav-content">
        <ul>
            <li>
                <a href="/">
                    <i class="iconfont icon-shouye1"></i>
                    <span>主页</span>
                </a>
            </li>
            <li>
                <a href="/tags">
                    <i class="iconfont icon-biaoqian1"></i>
                    <span>标签</span>
                </a>
            </li>
            <li>
                <a href="/archive">
                    <i class="iconfont icon-guidang2"></i>
                    <span>存档</span>
                </a>
            </li>
            <li>
                <a href="/about/">
                    <i class="iconfont icon-guanyu2"></i>
                    <span>关于</span>
                </a>
            </li>
            
            <li>
                <a id="search">
                    <i class="iconfont icon-sousuo1"></i>
                    <span>搜索</span>
                </a>
            </li>
            
        </ul>
    </div>
    
        <div id="toc" class="toc-article">
    <ol class="toc"><li class="toc-item toc-level-1"><a class="toc-link" href="#inotify-rsync"><span class="toc-text">inotify+rsync</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#inotify-自定义复制脚本"><span class="toc-text">inotify+自定义复制脚本</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#增量双写，diff复制策略（实操方案）"><span class="toc-text">增量双写，diff复制策略（实操方案）</span></a></li></ol>
</div>
    
</div>


<div class="search-field" id="search-field">
    <div class="search-container">
        <div class="search-input">
            <span id="esc-search"> <i class="icon-fanhui iconfont"></i></span>
            <input id="search-input">
            <span id="begin-search">搜索</span>
        </div>
        <div style="padding:10px 20px;font=10;">
            若搜索无反应请更换浏览器~
        </div>
        <div class="search-result-container" id="search-result-container">

        </div>
    </div>
</div>
        <div class="index-about-mobile">
            <i> 心若水， 自非凡。该博客已迁移至 https://www.cnblogs.com/xrszff </i>
        </div>
    </div>
    
    <div class="index-middle">
        <!-- Main Content -->
        


<div class="post-container">
    <div class="post-title">
        TB级NFS数据平滑迁移方案设计与实现
    </div>

    <div class="post-meta">
        <span class="attr">发布于：<span>2019-03-28 21:58:23</span></span>
        
        <span class="attr">标签：/
        
        <a class="tag" href="/tags/#分布式文件系统" title="分布式文件系统">分布式文件系统</a>
        <span>/</span>
        
        
        </span>
        <span class="attr">访问：<span id="busuanzi_value_page_pv"></span>
</span>

    </div>
    <div class="post-content no-indent">
        <p>前两种方案，是最开始调研的网上大多数人使用的方案修改出来的，但是实测后发现，对超大目录的同步非常无力，光inotify就要扫很久，并且扫完后，对新文件的检测也会出现问题。故最后是用最后一种方案，修改了老nfs服务的日志级别，从中捞取文件变化，引入延迟同步+diff无则同步策略。</p>
<p>下文出现ccdbfs的地方代表老nfs服务，mfs代表新nfs服务。</p>
<h1 id="inotify-rsync"><a href="#inotify-rsync" class="headerlink" title="inotify+rsync"></a>inotify+rsync</h1><p>对于文件里量不大的目录，复制起来快，且代码量少。</p>
<p>但是对超大目录，扫码起来无力。且事件感知会出现重复。</p>
<p><code>copy_ccdbfs_update_to_mfs.sh</code><br><figure class="highlight mel"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br></pre></td><td class="code"><pre><span class="line"># rsync auto sync script with inotify</span><br><span class="line"># <span class="number">2019</span>/<span class="number">03</span>/<span class="number">21</span></span><br><span class="line"></span><br><span class="line"># configs</span><br><span class="line">source_path=/mnt/ccdbfs/</span><br><span class="line">target_path=/mnt/mfs/</span><br><span class="line">rsync_bin=/usr/bin/rsync</span><br><span class="line">INOTIFY_EXCLUDE=<span class="string">'aladata.*'</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">cd $(<span class="keyword">dirname</span> $0)</span><br><span class="line"></span><br><span class="line">RSYNC_EXCLUDE=$(<span class="keyword">pwd</span>)/rsync_exclude.lst</span><br><span class="line"></span><br><span class="line">current_date=$(<span class="keyword">date</span> +%Y-%m-%d_%H%M%S)</span><br><span class="line">export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(<span class="keyword">pwd</span>)</span><br><span class="line"></span><br><span class="line">log_file=$(<span class="keyword">pwd</span>)/runcopy.<span class="keyword">log</span>.$&#123;current_date&#125;</span><br><span class="line">inotifywait_bin=$(<span class="keyword">pwd</span>)/inotifywait</span><br><span class="line"></span><br><span class="line">inotify_fun()&#123;</span><br><span class="line">    $&#123;inotifywait_bin&#125; -mrq --timefmt <span class="string">'%Y/%m/%d-%H:%M:%S'</span> --<span class="keyword">format</span> <span class="string">'%T %w %f'</span> \</span><br><span class="line">     --exclude $&#123;INOTIFY_EXCLUDE&#125; -e modify,create,<span class="keyword">move</span>,attrib $&#123;source_path&#125; \</span><br><span class="line">    | <span class="keyword">while</span> read <span class="keyword">file</span></span><br><span class="line">        <span class="keyword">do</span></span><br><span class="line">            $&#123;rsync_bin&#125; -auvrtzopgP --exclude-from=$&#123;RSYNC_EXCLUDE&#125; --progress --bwlimit=<span class="number">800000</span> $&#123;source_path&#125; $&#123;target_path&#125;</span><br><span class="line">        done</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">#inotify <span class="keyword">log</span></span><br><span class="line">inotify_fun &amp;&gt; $&#123;log_file&#125; &amp;</span><br></pre></td></tr></table></figure></p>
<h1 id="inotify-自定义复制脚本"><a href="#inotify-自定义复制脚本" class="headerlink" title="inotify+自定义复制脚本"></a>inotify+自定义复制脚本</h1><p>inotify启一个光监控文件新增修改事件的脚本，导入到change_file.list。</p>
<p>自定义复制脚本去读取并且逻辑控制后复制。<br><code>inotify_ccdbfs_change.sh</code><br><figure class="highlight mel"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"># find ccdbfs change with inotify, export to <span class="keyword">file</span></span><br><span class="line"># <span class="number">2019</span>/<span class="number">03</span>/<span class="number">21</span></span><br><span class="line"></span><br><span class="line"># configs</span><br><span class="line">source_path=/mnt/ccdbfs/</span><br><span class="line">INOTIFY_EXCLUDE=<span class="string">'.*(\.swx|\.swp)$|aladata.*'</span></span><br><span class="line"></span><br><span class="line">cd $(<span class="keyword">dirname</span> $0)</span><br><span class="line">current_date=$(<span class="keyword">date</span> +%Y-%m-%d_%H%M%S)</span><br><span class="line">export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(<span class="keyword">pwd</span>)</span><br><span class="line">inotifywait_bin=$(<span class="keyword">pwd</span>)/inotifywait</span><br><span class="line"></span><br><span class="line">$&#123;inotifywait_bin&#125; -mrq --timefmt <span class="string">'%Y/%m/%d-%H:%M:%S'</span> --<span class="keyword">format</span> <span class="string">'%w%f %e %T'</span> \</span><br><span class="line">    --exclude $&#123;INOTIFY_EXCLUDE&#125; -e create,close_write $&#123;source_path&#125; \</span><br><span class="line">| <span class="keyword">while</span> read line</span><br><span class="line">    <span class="keyword">do</span></span><br><span class="line">        echo $&#123;line&#125; &gt;&gt; $(<span class="keyword">pwd</span>)/change_file.list</span><br><span class="line">    done &amp;</span><br></pre></td></tr></table></figure></p>
<p><code>change_file.list</code><br><figure class="highlight groovy"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="regexp">/mnt/</span>mfs<span class="regexp">/dirtest4 CREATE,ISDIR 2019/</span><span class="number">03</span>/<span class="number">21</span><span class="number">-22</span>:<span class="number">30</span>:<span class="number">06</span></span><br><span class="line"><span class="regexp">/mnt/</span>mfs<span class="regexp">/dirtest4/</span><span class="number">5</span> CREATE,ISDIR <span class="number">2019</span><span class="regexp">/03/</span><span class="number">21</span><span class="number">-22</span>:<span class="number">30</span>:<span class="number">06</span></span><br><span class="line"><span class="regexp">/mnt/</span>mfs<span class="regexp">/a.test CLOSE_WRITE,CLOSE 2019/</span><span class="number">03</span>/<span class="number">21</span><span class="number">-22</span>:<span class="number">30</span>:<span class="number">17</span></span><br><span class="line"><span class="regexp">/mnt/</span>mfs<span class="regexp">/c.test CLOSE_WRITE,CLOSE 2019/</span><span class="number">03</span>/<span class="number">21</span><span class="number">-22</span>:<span class="number">30</span>:<span class="number">24</span></span><br><span class="line"><span class="regexp">/mnt/</span>mfs<span class="regexp">/aa CREATE 2019/</span><span class="number">03</span>/<span class="number">21</span><span class="number">-22</span>:<span class="number">30</span>:<span class="number">45</span></span><br><span class="line"><span class="regexp">/mnt/</span>mfs<span class="regexp">/aa CLOSE_WRITE,CLOSE 2019/</span><span class="number">03</span>/<span class="number">21</span><span class="number">-22</span>:<span class="number">30</span>:<span class="number">45</span></span><br></pre></td></tr></table></figure></p>
<p><code>inotify_ccdbfs_change_cp.py</code><br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># -*- coding: utf-8 -*-</span></span><br><span class="line"><span class="comment"># 持续读取inotify_ccdbfs_change.sh监控到的变化文件change_file.list，从ccdbfs同步到mfs</span></span><br><span class="line"><span class="comment"># 2019/03/22</span></span><br><span class="line"><span class="keyword">import</span> os</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">SOURCE_ROOT = <span class="string">'/mnt/ccdbfs'</span></span><br><span class="line">TARGET_ROOT = <span class="string">'/mnt/mfs'</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">deal_ori_change_file_list</span><span class="params">()</span>:</span></span><br><span class="line">    <span class="string">'''获取change_file.list文件得到变化文件list</span></span><br><span class="line"><span class="string">       顺便追加change_file.list到change_file.bak</span></span><br><span class="line"><span class="string">       然后清空change_file.list</span></span><br><span class="line"><span class="string">    '''</span></span><br><span class="line">    change_file_path = os.getcwd() + <span class="string">'/change_file.list'</span></span><br><span class="line">    change_file_back_path = os.getcwd() + <span class="string">'/change_file.bak'</span></span><br><span class="line">    <span class="keyword">with</span> open(change_file_path) <span class="keyword">as</span> f:</span><br><span class="line">        row_list = f.read().splitlines()</span><br><span class="line">    <span class="comment"># 追加备份</span></span><br><span class="line">    os.system(<span class="string">'echo "" &gt;&gt; '</span> + change_file_back_path)  <span class="comment"># 相当于换行</span></span><br><span class="line">    os.system(<span class="string">'cat '</span> + change_file_path + <span class="string">' &gt;&gt; '</span> + change_file_back_path)</span><br><span class="line"></span><br><span class="line">    <span class="comment"># 清空原change_file.list</span></span><br><span class="line">    os.system(<span class="string">'cat /dev/null &gt; '</span> + change_file_path)</span><br><span class="line"></span><br><span class="line">    <span class="keyword">return</span> row_list</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">uniq_row_list</span><span class="params">(row_list)</span>:</span></span><br><span class="line">    <span class="string">'''对获取到的原始row_list拆分和保序去重</span></span><br><span class="line"><span class="string">       返回 dir_list, file_list</span></span><br><span class="line"><span class="string">    '''</span></span><br><span class="line">    dir_list = []</span><br><span class="line">    file_list = []</span><br><span class="line"></span><br><span class="line">    <span class="keyword">for</span> row <span class="keyword">in</span> row_list:</span><br><span class="line">        <span class="keyword">try</span>:</span><br><span class="line">            obj, effect, _ = row.split(<span class="string">' '</span>)</span><br><span class="line">            <span class="keyword">if</span> <span class="string">'ISDIR'</span> <span class="keyword">in</span> effect:</span><br><span class="line">                <span class="keyword">if</span> obj <span class="keyword">not</span> <span class="keyword">in</span> dir_list:</span><br><span class="line">                    dir_list.append(obj)</span><br><span class="line">            <span class="keyword">else</span>:</span><br><span class="line">                <span class="keyword">if</span> obj <span class="keyword">not</span> <span class="keyword">in</span> file_list:</span><br><span class="line">                    file_list.append(obj)</span><br><span class="line">        <span class="keyword">except</span>:</span><br><span class="line">            print(str(time.time()) + <span class="string">'uniq_row_list except'</span>)</span><br><span class="line"></span><br><span class="line">    <span class="keyword">return</span> dir_list, file_list</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">copy_change</span><span class="params">(dir_list, file_list)</span>:</span></span><br><span class="line">    <span class="string">'''在mfs中复制ccdbfs中的变化</span></span><br><span class="line"><span class="string">       先复制创建目录，再复制</span></span><br><span class="line"><span class="string">    '''</span></span><br><span class="line">    <span class="comment"># 优先在mfs中创建没有的目录，再去复制文件</span></span><br><span class="line">    <span class="keyword">for</span> a_dir <span class="keyword">in</span> dir_list:</span><br><span class="line">        dst = a_dir.replace(SOURCE_ROOT, TARGET_ROOT)</span><br><span class="line">        <span class="keyword">if</span> os.path.exists(dst) == <span class="literal">False</span>:</span><br><span class="line">            os.system(<span class="string">'mkdir -p '</span> + dst)</span><br><span class="line">    </span><br><span class="line">    <span class="keyword">for</span> a_file <span class="keyword">in</span> file_list:</span><br><span class="line">        dst_file = a_file.replace(SOURCE_ROOT, TARGET_ROOT)</span><br><span class="line">        <span class="comment"># 防止目标地址目录没创建</span></span><br><span class="line">        (file_path,file_name) = os.path.split(dst_file)</span><br><span class="line">        <span class="keyword">if</span> os.path.exists(file_path) == <span class="literal">False</span>:</span><br><span class="line">            os.system(<span class="string">'mkdir -p '</span> + file_path)</span><br><span class="line">        <span class="comment"># 复制文件或者软链接</span></span><br><span class="line">        os.system(<span class="string">'cp -d '</span> + a_file + <span class="string">' '</span> + dst_file)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line">    last_time = <span class="number">0</span></span><br><span class="line">    <span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line">        change_file_path = os.getcwd() + <span class="string">'/change_file.list'</span></span><br><span class="line">        change_file_back_path = os.getcwd() + <span class="string">'/change_file.bak'</span></span><br><span class="line">        <span class="comment"># 获取change_file.list文件得到变化文件list</span></span><br><span class="line">        <span class="keyword">with</span> open(change_file_path) <span class="keyword">as</span> f:</span><br><span class="line">            row_list = f.read().splitlines()</span><br><span class="line"></span><br><span class="line">        <span class="comment"># 执行备份条件是 变化记录满100条 或 间隔5s且有变化记录</span></span><br><span class="line">        <span class="comment"># 所以当没满100条，且没满5s的时候，或者满了5s但是没有变化记录，不进行后面的备份</span></span><br><span class="line">        <span class="keyword">if</span> len(row_list) &lt; <span class="number">100</span>:</span><br><span class="line">            <span class="keyword">if</span> time.time() - last_time &lt; <span class="number">5</span>:</span><br><span class="line">                print(<span class="string">'row_list &lt; 100 and delta_time &lt; 5'</span>)</span><br><span class="line">                time.sleep(<span class="number">1</span>)</span><br><span class="line">                <span class="keyword">continue</span></span><br><span class="line">            <span class="keyword">elif</span> len(row_list) == <span class="number">0</span>:</span><br><span class="line">                print(<span class="string">'delta_time &gt; 5 but row_list == 0'</span>)</span><br><span class="line">                time.sleep(<span class="number">1</span>)</span><br><span class="line">                <span class="keyword">continue</span></span><br><span class="line"></span><br><span class="line">        <span class="comment"># 满足条件，开始备份</span></span><br><span class="line">        print(<span class="string">'cp start'</span>)</span><br><span class="line">        last_time = time.time()</span><br><span class="line">        <span class="comment"># 追加备份历史记录</span></span><br><span class="line">        os.system(<span class="string">'echo "" &gt;&gt; '</span> + change_file_back_path)  <span class="comment"># 相当于换行</span></span><br><span class="line">        os.system(<span class="string">'cat '</span> + change_file_path + <span class="string">' &gt;&gt; '</span> + change_file_back_path)</span><br><span class="line">        <span class="comment"># 清空原change_file.list</span></span><br><span class="line">        os.system(<span class="string">'cat /dev/null &gt; '</span> + change_file_path)</span><br><span class="line"></span><br><span class="line">        dir_list, file_list = uniq_row_list(row_list)</span><br><span class="line">        print(<span class="string">'dir_list: '</span>, dir_list)</span><br><span class="line">        print(<span class="string">'file_list: '</span>, file_list)</span><br><span class="line">        copy_change(dir_list, file_list)</span><br><span class="line">        time.sleep(<span class="number">1</span>)</span><br></pre></td></tr></table></figure></p>
<h1 id="增量双写，diff复制策略（实操方案）"><a href="#增量双写，diff复制策略（实操方案）" class="headerlink" title="增量双写，diff复制策略（实操方案）"></a>增量双写，diff复制策略（实操方案）</h1><p>捞取老nfs日志+自定义双写脚本 </p>
<p><code>cp_ccdbfs_change.py</code> 增量更新同步双写脚本。集群业务机器，有对老ccdbfs写入的都启一个。</p>
<p>对于捞取到的文件变化日志进行双重筛选判断,因：</p>
<ul>
<li>测试发现，复制没问题，但是清理复制过的记录的时候，要去查询一下文件的修改时间，又产生了一条读的操作，还是一模一样的日志，又会再删除后新添加进全局map去，所以这样的话就是无限循环删不掉。</li>
<li>打算修改方案，在复制后，不把那条记录删除，而是把存的log时间改为0，当轮询线程取出来的记录，时间为0，直接跳过，不去查询修改时间了。</li>
<li>之后如果该文件再次修改，触发了日志后，会看如果时间为0，证明上次复制过了，但是现在又新修改了，那么重新刷成触发日志的新时间。</li>
<li>顺便，如果是目录，时间存成1，也不再处理。</li>
<li>内存应该撑得住，因为存储的map只是文件的信息而不是文件内容，并且，经常有进入目录或者查询修改时间产生的日志记录。</li>
<li>上面这样的话，目录可以排除了，但是对于文件，还是会因为查询修改时间，产生日志记录，并且刷新log_time。所以还应该设置一个检查过的map记录（值为上一次复制或检查完成的时间），两个结合判断。</li>
</ul>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br><span class="line">182</span><br><span class="line">183</span><br><span class="line">184</span><br><span class="line">185</span><br><span class="line">186</span><br><span class="line">187</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># -*- coding: utf-8 -*-</span></span><br><span class="line"><span class="comment"># 持续读取从ccdbfs日志捞取出来的变更文件，并同步到mfs对应路径下</span></span><br><span class="line"><span class="comment"># 2019/03/26</span></span><br><span class="line"><span class="keyword">import</span> os</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="keyword">import</span> sys</span><br><span class="line"><span class="keyword">import</span> subprocess</span><br><span class="line"><span class="keyword">import</span> threading</span><br><span class="line"></span><br><span class="line">CCDBFS_LOG_PATH = <span class="string">'/home/work/nfs_client/log/Client.log'</span></span><br><span class="line">SOURCE_ROOT = <span class="string">'/mnt/ccdbfs'</span></span><br><span class="line">TARGET_ROOT = <span class="string">'/mnt/mfs'</span></span><br><span class="line">CCDBFS_SAVE_ROOT = <span class="string">'/disk'</span></span><br><span class="line">CCDBFS_SEARCH_TOOLS_ROOT = <span class="string">'/home/work/copy_ccdbfs_to_mfs/output'</span></span><br><span class="line"></span><br><span class="line">EXCLUDE_LIST = [<span class="string">'/mnt/ccdbfs/aladata'</span>]</span><br><span class="line"></span><br><span class="line">DELAY_DEAL_TIME = <span class="number">120</span></span><br><span class="line"></span><br><span class="line">MAP_DICT = &#123;&#125;  <span class="comment"># 存储有变更的文件信息</span></span><br><span class="line">INODE_PATH_DICT = &#123;&#125;  <span class="comment"># 提供存储和查询inode对应路径用</span></span><br><span class="line">FINISHED_DICT = &#123;&#125;  <span class="comment"># 存储复制过的文件信息，值记录上次检查完成时间，值为1表示不再做检查</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">myprint</span><span class="params">(*args)</span>:</span></span><br><span class="line">    ltime = time.localtime(time.time())</span><br><span class="line">    now_time = time.strftime(<span class="string">"%Y-%m-%d %H:%M:%S, "</span>, ltime)        </span><br><span class="line">    print(now_time, args)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">copy_change</span><span class="params">(obj_path)</span>:</span></span><br><span class="line">    <span class="string">'''在mfs中复制ccdbfs中的变化</span></span><br><span class="line"><span class="string">    '''</span></span><br><span class="line">    <span class="keyword">try</span>:</span><br><span class="line">        target_path = obj_path.replace(SOURCE_ROOT, TARGET_ROOT)</span><br><span class="line">        <span class="comment"># 先在mfs目标路径下判断路径是否存在，没有则创建</span></span><br><span class="line">        (file_path,file_name) = os.path.split(target_path)</span><br><span class="line">        <span class="keyword">if</span> os.path.exists(file_path) == <span class="literal">False</span>:</span><br><span class="line">            os.system(<span class="string">'mkdir -p '</span> + file_path)</span><br><span class="line"></span><br><span class="line">        <span class="comment"># 复制文件或者软链接</span></span><br><span class="line">        <span class="comment"># -p 修改时间和访问权限也复制到新文件</span></span><br><span class="line">        <span class="comment"># -d 复制时保留链接</span></span><br><span class="line">        <span class="comment"># -u 源文件修改时间更新才复制</span></span><br><span class="line">        os.system(<span class="string">"""sh -c 'nohup cp -pdu "%s" "%s" &amp;'"""</span> % (obj_path, target_path))</span><br><span class="line">    <span class="keyword">except</span>:</span><br><span class="line">        myprint(<span class="string">'cp change error: '</span> + obj_path)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">extract_dir_objname</span><span class="params">(log_buff)</span>:</span></span><br><span class="line">    <span class="string">'''从日志语句中提取出父目录以及存储对象名</span></span><br><span class="line"><span class="string">    '''</span></span><br><span class="line">    <span class="keyword">try</span>:</span><br><span class="line">        tmplist = log_buff.split(<span class="string">' '</span>)</span><br><span class="line">        log_time = <span class="string">'2019-'</span> + tmplist[<span class="number">1</span>] + <span class="string">' '</span> + tmplist[<span class="number">2</span>]</span><br><span class="line">        log_time_stamp = time.mktime(time.strptime(log_time, <span class="string">"%Y-%m-%d %H:%M:%S:"</span>))</span><br><span class="line">        <span class="comment"># 去掉 parentDirInode:0xa0000000101b9c32, 的多余字符</span></span><br><span class="line">        dir_inode = tmplist[<span class="number">8</span>].replace(<span class="string">'parentDirInode:'</span>, <span class="string">''</span>).replace(<span class="string">','</span>, <span class="string">''</span>)</span><br><span class="line">        <span class="comment"># 只取出存储对象名字</span></span><br><span class="line">        objname = tmplist[<span class="number">9</span>].replace(<span class="string">'\n'</span>,<span class="string">''</span>).split(<span class="string">':'</span>)[<span class="number">1</span>]</span><br><span class="line">        <span class="keyword">return</span> log_time_stamp, dir_inode, objname</span><br><span class="line">    <span class="keyword">except</span>:</span><br><span class="line">        myprint(<span class="string">'extract_dir_objname error, '</span> + log_buff)</span><br><span class="line">        <span class="keyword">return</span> <span class="literal">None</span>, <span class="literal">None</span>, <span class="literal">None</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">tail_log_add_map</span><span class="params">()</span>:</span></span><br><span class="line">    <span class="string">'''</span></span><br><span class="line"><span class="string">    Desc: 持续捞取文件变更日志，并将满足条件的变更存入全局MAP_DICT</span></span><br><span class="line"><span class="string">    '''</span></span><br><span class="line">    <span class="comment"># 持续捞取长这样的日志，必须包含 ]Lookup, 有逗号</span></span><br><span class="line">    <span class="comment"># DE<span class="doctag">BUG:</span> 03-26 11:45:56:  Client * 15176 [ccdb:NFSClient.cpp:625:15176]Lookup, parentDirInode:0xa0000001198bee40, dentryName:filelinktest</span></span><br><span class="line">    p = subprocess.Popen(<span class="string">'tail -F '</span> + CCDBFS_LOG_PATH, shell=<span class="literal">True</span>, stdout=subprocess.PIPE)</span><br><span class="line">    <span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line">        buff = p.stdout.readline()</span><br><span class="line">        <span class="comment"># 克服直接在Popen中grep有问题的办法</span></span><br><span class="line">        <span class="keyword">if</span> <span class="string">']Lookup,'</span> <span class="keyword">not</span> <span class="keyword">in</span> buff:</span><br><span class="line">            <span class="keyword">continue</span></span><br><span class="line">        <span class="keyword">if</span> buff == <span class="string">''</span> <span class="keyword">and</span> p.poll() != <span class="literal">None</span>:</span><br><span class="line">            <span class="keyword">break</span></span><br><span class="line">        </span><br><span class="line">        log_time_stamp, dir_inode, objname = extract_dir_objname(buff)</span><br><span class="line">        <span class="keyword">if</span> (log_time_stamp == <span class="literal">None</span>) <span class="keyword">or</span> (dir_inode == <span class="literal">None</span>) <span class="keyword">or</span> (objname == <span class="literal">None</span>):</span><br><span class="line">            <span class="keyword">continue</span></span><br><span class="line"></span><br><span class="line">        <span class="comment"># 若不存在，添加之</span></span><br><span class="line">        map_value = MAP_DICT.get(dir_inode + <span class="string">'+'</span> + objname)</span><br><span class="line">        <span class="keyword">if</span>  map_value == <span class="literal">None</span>:</span><br><span class="line">            MAP_DICT[dir_inode + <span class="string">'+'</span> + objname] = log_time_stamp</span><br><span class="line">        <span class="keyword">else</span>:</span><br><span class="line">            last_finished_time = FINISHED_DICT.get(dir_inode + <span class="string">'+'</span> + objname)</span><br><span class="line">            <span class="comment"># 之前复制完成log_time变成了0，再次有修改才变换这里的log_time</span></span><br><span class="line">            <span class="comment"># 上次检测或者复制完成10s内，视为读操作，不修改log_time</span></span><br><span class="line">            <span class="keyword">if</span> last_finished_time == <span class="literal">None</span>:</span><br><span class="line">                <span class="comment"># FINISHED_DICT无记录，表示要去检测</span></span><br><span class="line">                last_finished_time = <span class="number">0</span></span><br><span class="line">            <span class="keyword">if</span> (map_value == <span class="number">0</span>) <span class="keyword">and</span> (time.time() - last_finished_time &gt; <span class="number">10</span>):</span><br><span class="line">                    MAP_DICT[dir_inode + <span class="string">'+'</span> + objname] = log_time_stamp</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">check_map_dict</span><span class="params">()</span>:</span></span><br><span class="line">    <span class="string">'''</span></span><br><span class="line"><span class="string">    Desc: 轮询遍历MAP_DICT将满足条件的文件进行复制或删除记录</span></span><br><span class="line"><span class="string">    '''</span></span><br><span class="line">    <span class="keyword">global</span> MAP_DICT</span><br><span class="line">    <span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line">        <span class="keyword">for</span> k, log_time <span class="keyword">in</span> MAP_DICT.items():</span><br><span class="line">            <span class="comment"># myprint(k, log_time)</span></span><br><span class="line">            <span class="comment"># log_time为0表示上一次复制或检查完成，log_time为1表示是目录永远不再检查，不用操作</span></span><br><span class="line">            <span class="comment"># 在复制文件前会检查，如果目录不存在会创建</span></span><br><span class="line">            <span class="keyword">if</span> log_time == <span class="number">0</span> <span class="keyword">or</span> log_time == <span class="number">1</span>:</span><br><span class="line">                <span class="keyword">continue</span></span><br><span class="line">            dir_inode, objname = k.split(<span class="string">'+'</span>)</span><br><span class="line">            <span class="comment"># 先查全局INODE_PATH_DICT中存了路径没，没有再去查ccdbfs客户端</span></span><br><span class="line">            dir_path = INODE_PATH_DICT.get(dir_inode)</span><br><span class="line">            <span class="keyword">if</span> dir_path == <span class="literal">None</span>:</span><br><span class="line">                <span class="keyword">try</span>:</span><br><span class="line">                    op = os.popen(<span class="string">"cd %s &amp;&amp; echo 'lookup %s' | ./bin/Cli -x"</span> % (CCDBFS_SEARCH_TOOLS_ROOT, dir_inode))</span><br><span class="line">                    ret = op.read()</span><br><span class="line">                    <span class="comment"># ccdbfs客户端存的路径是/disk开头，挂载路径是/mnt/ccdbfs</span></span><br><span class="line">                    dir_path = ret.split(<span class="string">' '</span>)[<span class="number">5</span>].replace(CCDBFS_SAVE_ROOT, SOURCE_ROOT)</span><br><span class="line">                    INODE_PATH_DICT[dir_inode] = dir_path</span><br><span class="line">                    myprint(<span class="string">'search ccdbfs client:%s-&gt;%s'</span> % (dir_inode, dir_path))</span><br><span class="line">                <span class="keyword">except</span>:</span><br><span class="line">                    myprint(<span class="string">'search dir path error: '</span> + k)</span><br><span class="line">                    <span class="comment"># 若获取不到路径，跳过这次复制</span></span><br><span class="line">                    MAP_DICT[k] = <span class="number">0</span></span><br><span class="line">                    FINISHED_DICT[k] = time.time()</span><br><span class="line">                    <span class="keyword">continue</span></span><br><span class="line"></span><br><span class="line">            <span class="comment"># 拼接文件路径</span></span><br><span class="line">            obj_path = dir_path + objname</span><br><span class="line"></span><br><span class="line">            <span class="comment"># 若obj_path本身是目录，不用处理</span></span><br><span class="line">            <span class="keyword">if</span> os.path.isdir(obj_path) == <span class="literal">True</span>:</span><br><span class="line">                <span class="comment"># 修改log_time为1，当做标记用</span></span><br><span class="line">                MAP_DICT[k] = <span class="number">1</span></span><br><span class="line">                <span class="keyword">continue</span></span><br><span class="line"></span><br><span class="line">            <span class="comment"># 排除一些不需要监控的目录</span></span><br><span class="line">            exclude_flag = <span class="literal">False</span></span><br><span class="line">            <span class="keyword">for</span> exclude <span class="keyword">in</span> EXCLUDE_LIST:</span><br><span class="line">                <span class="keyword">if</span> exclude <span class="keyword">in</span> dir_path:</span><br><span class="line">                    exclude_flag = <span class="literal">True</span></span><br><span class="line">                    <span class="keyword">break</span></span><br><span class="line">            <span class="keyword">if</span> exclude_flag == <span class="literal">True</span>:</span><br><span class="line">                MAP_DICT[k] = <span class="number">1</span>  <span class="comment"># 故意置1，永不检查</span></span><br><span class="line">                <span class="keyword">continue</span></span><br><span class="line"></span><br><span class="line">            <span class="comment"># 若是临时文件，不存在了，跳过这次复制</span></span><br><span class="line">            <span class="keyword">if</span> os.path.exists(obj_path) == <span class="literal">False</span>:</span><br><span class="line">                myprint(obj_path + <span class="string">' is inexistent'</span>)</span><br><span class="line">                MAP_DICT[k] = <span class="number">0</span></span><br><span class="line">                FINISHED_DICT[k] = time.time()</span><br><span class="line">                <span class="keyword">continue</span></span><br><span class="line"></span><br><span class="line">            <span class="comment"># 获取最新修改时间</span></span><br><span class="line">            <span class="keyword">try</span>:</span><br><span class="line">                modify_time = os.stat(obj_path).st_mtime</span><br><span class="line">            <span class="keyword">except</span>:</span><br><span class="line">                myprint(<span class="string">'os.stat().st_mtime error: '</span> + obj_path)</span><br><span class="line">                <span class="keyword">continue</span></span><br><span class="line">            now_time = time.time()</span><br><span class="line">            <span class="comment"># -5秒的判断，是防止极短时间写入后，log日志时间稍大于修改时间。</span></span><br><span class="line">            <span class="keyword">if</span> now_time - modify_time &gt; DELAY_DEAL_TIME <span class="keyword">and</span> modify_time - log_time &gt;= <span class="number">-5</span>:</span><br><span class="line">                <span class="comment"># 复制同步，并修改键值表示复制过</span></span><br><span class="line">                myprint(<span class="string">'cp and set MAP_DICT[k]=0, FINISHED_DICT=now_time, '</span> + obj_path)</span><br><span class="line">                copy_change(obj_path)</span><br><span class="line">                MAP_DICT[k] = <span class="number">0</span></span><br><span class="line">                FINISHED_DICT[k] = now_time</span><br><span class="line">            <span class="keyword">else</span>:</span><br><span class="line">                <span class="comment"># 也不能让MAP_DICT野蛮增长，对于读操作（日志时间大于修改时间5s以上）的log，要清除掉</span></span><br><span class="line">                <span class="keyword">if</span> now_time - modify_time &gt; DELAY_DEAL_TIME <span class="keyword">and</span> log_time - modify_time &gt; <span class="number">5</span>:</span><br><span class="line">                    <span class="comment"># 修改键值表示检查过</span></span><br><span class="line">                    myprint(<span class="string">'no change, set MAP_DICT[k]=0, FINISHED_DICT=now_time, '</span> + obj_path)</span><br><span class="line">                    MAP_DICT[k] = <span class="number">0</span></span><br><span class="line">                    FINISHED_DICT[k] = now_time</span><br><span class="line">            <span class="comment"># time.sleep(2)</span></span><br><span class="line">        time.sleep(<span class="number">60</span>)</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line">    t1 = threading.Thread(target = tail_log_add_map)</span><br><span class="line">    t2 = threading.Thread(target = check_map_dict)</span><br><span class="line"></span><br><span class="line">    t1.start()</span><br><span class="line">    t2.start()</span><br><span class="line"></span><br><span class="line">    t1.join()</span><br><span class="line">    t2.join()</span><br></pre></td></tr></table></figure>
<p><code>copy_inexistent_file.py</code> 同步剩余全量文件，若无则同策略。另起一台机器连接新老nfs客户端即可启动之。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># -*- coding: utf-8 -*-</span></span><br><span class="line"><span class="comment"># 平台nfs未被inotify检测到的文件迁移</span></span><br><span class="line"><span class="comment"># 2019/03/18</span></span><br><span class="line"><span class="keyword">import</span> os</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"></span><br><span class="line">DIRS_LIST = []  <span class="comment"># 记录老文件夹下不断遍历添加进来的目录名，检测完后抛出，当队列使用</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># SOURCE_ROOT，TARGET_ROOT提供程序中替换目录路径用</span></span><br><span class="line"><span class="comment"># 为防止其他文件也用到ccdbfs字眼，所以全部执行完后</span></span><br><span class="line"><span class="comment"># 手动修改新mfs目录里ccdbfs软链接，而不再程序中特殊控制</span></span><br><span class="line">SOURCE_ROOT = <span class="string">'/mnt/ccdbfs'</span></span><br><span class="line">TARGET_ROOT = <span class="string">'/mnt/mfs'</span></span><br><span class="line"><span class="comment"># SOURCE_ROOT = '/mnt/srcdir'</span></span><br><span class="line"><span class="comment"># TARGET_ROOT = '/mnt/tardir'</span></span><br><span class="line"></span><br><span class="line">EXCLUDE_LIST = [<span class="string">'/mnt/ccdbfs/aladata'</span>]</span><br><span class="line"><span class="comment"># EXCLUDE_LIST = ['/mnt/srcdir/aladata']</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">myprint</span><span class="params">(*args)</span>:</span></span><br><span class="line">    ltime = time.localtime(time.time())</span><br><span class="line">    now_time = time.strftime(<span class="string">"%Y-%m-%d %H:%M:%S, "</span>, ltime)        </span><br><span class="line">    print(now_time, args)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">dir_info</span><span class="params">(file_dir)</span>:</span></span><br><span class="line">    <span class="keyword">for</span> root, dirs, files <span class="keyword">in</span> os.walk(file_dir):</span><br><span class="line">        <span class="comment"># myprint('root_dir:', root)  # 当前目录路径</span></span><br><span class="line">        <span class="comment"># myprint('sub_dirs:', dirs)  # 当前路径下所有子目录</span></span><br><span class="line">        <span class="comment"># myprint('files:', files)  # 当前路径下所有非目录子文件</span></span><br><span class="line">        <span class="comment"># 补全目录全路径</span></span><br><span class="line">        full_path_dirs = [root + <span class="string">'/'</span> + i <span class="keyword">for</span> i <span class="keyword">in</span> dirs]</span><br><span class="line">        full_path_dirs_res = []</span><br><span class="line"></span><br><span class="line">        <span class="comment"># 过滤某些不拷贝的目录</span></span><br><span class="line">        <span class="keyword">for</span> i <span class="keyword">in</span> full_path_dirs:</span><br><span class="line">            exclude_flat = <span class="literal">False</span></span><br><span class="line">            <span class="keyword">for</span> exclude <span class="keyword">in</span> EXCLUDE_LIST:</span><br><span class="line">                <span class="keyword">if</span> exclude <span class="keyword">in</span> i:</span><br><span class="line">                    exclude_flat = <span class="literal">True</span></span><br><span class="line">                    <span class="keyword">break</span></span><br><span class="line">            <span class="keyword">if</span> exclude_flat == <span class="literal">False</span>:</span><br><span class="line">                full_path_dirs_res.append(i)</span><br><span class="line"></span><br><span class="line">        <span class="comment"># 补全文件全路径</span></span><br><span class="line">        full_path_files = [root + <span class="string">'/'</span> + i <span class="keyword">for</span> i <span class="keyword">in</span> files]</span><br><span class="line">        <span class="keyword">return</span> full_path_dirs_res, full_path_files</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">doing_copy_file</span><span class="params">(dir_path)</span>:</span></span><br><span class="line">    <span class="string">'''传入原ccdbfs目录下的某文件夹路径，复制其中文件</span></span><br><span class="line"><span class="string">       并将其中文件夹推入全局DIRS_LIST</span></span><br><span class="line"><span class="string">    '''</span></span><br><span class="line">    dirs, files = dir_info(dir_path)</span><br><span class="line">    <span class="comment"># 发现目录，推入DIRS_LIST</span></span><br><span class="line">    <span class="keyword">global</span> DIRS_LIST</span><br><span class="line">    DIRS_LIST += dirs</span><br><span class="line"></span><br><span class="line">    <span class="keyword">for</span> src_file <span class="keyword">in</span> files:</span><br><span class="line">        <span class="comment"># mfs下属目录中不存在该文件，才复制</span></span><br><span class="line">        dst_file = src_file.replace(SOURCE_ROOT, TARGET_ROOT)</span><br><span class="line">        <span class="keyword">if</span> os.path.exists(dst_file) == <span class="literal">False</span>:</span><br><span class="line">            <span class="keyword">try</span>:</span><br><span class="line">                <span class="comment"># 防止目标地址目录没创建</span></span><br><span class="line">                (file_path,file_name) = os.path.split(dst_file)</span><br><span class="line">                <span class="keyword">if</span> os.path.exists(file_path) == <span class="literal">False</span>:</span><br><span class="line">                    os.system(<span class="string">'mkdir -p '</span> + file_path)</span><br><span class="line">                <span class="comment"># 正式复制该文件</span></span><br><span class="line">                myprint(<span class="string">'cp -pdu "%s" "%s"'</span> % (src_file, dst_file))</span><br><span class="line">                os.system(<span class="string">'cp -pdu "%s" "%s"'</span> % (src_file, dst_file))</span><br><span class="line">            <span class="keyword">except</span>:</span><br><span class="line">                myprint(<span class="string">'copy file except: '</span> + dst_file)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">make_soft_link</span><span class="params">(soft_file)</span>:</span></span><br><span class="line">    <span class="string">'''将ccdbfs中的软链接复制到对应mfs目录相同层级下</span></span><br><span class="line"><span class="string">    '''</span></span><br><span class="line">    <span class="keyword">try</span>:</span><br><span class="line">        dst_file = soft_file.replace(SOURCE_ROOT, TARGET_ROOT)</span><br><span class="line">        <span class="keyword">if</span> os.path.exists(dst_file) == <span class="literal">True</span>:</span><br><span class="line">            <span class="comment"># 存在则不用重复复制</span></span><br><span class="line">            myprint(dst_file + <span class="string">'already exists'</span>)</span><br><span class="line">            <span class="keyword">return</span></span><br><span class="line">        <span class="comment"># 防止目标地址目录没创建</span></span><br><span class="line">        (file_path,file_name) = os.path.split(dst_file)</span><br><span class="line">        <span class="keyword">if</span> os.path.exists(file_path) == <span class="literal">False</span>:</span><br><span class="line">            os.system(<span class="string">'mkdir -p '</span> + file_path)</span><br><span class="line">        <span class="comment"># 正式复制该软链接</span></span><br><span class="line">        myprint(<span class="string">'cp -pdu "%s" "%s"'</span> % (soft_file, dst_file))</span><br><span class="line">        os.system(<span class="string">'cp -pdu "%s" "%s"'</span> % (soft_file, dst_file))</span><br><span class="line">    <span class="keyword">except</span>:</span><br><span class="line">        myprint(<span class="string">'make soft link except, old:'</span> + soft_file)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line">    <span class="comment"># 先把源头根目录下的文件复制，以及其文件夹推入DIRS_LIST</span></span><br><span class="line">    doing_copy_file(SOURCE_ROOT)</span><br><span class="line"></span><br><span class="line">    <span class="comment"># 一层一层目录遍历完，DIRS_LIST清空程序才结束</span></span><br><span class="line">    <span class="keyword">while</span> len(DIRS_LIST) &gt; <span class="number">0</span>:</span><br><span class="line">        current = DIRS_LIST.pop(<span class="number">0</span>)</span><br><span class="line">        <span class="comment"># myprint('now doing-&gt;', current)</span></span><br><span class="line">        <span class="keyword">if</span> os.path.islink(current) == <span class="literal">True</span>:</span><br><span class="line">            <span class="comment"># 若是软链接，不进入目录，只获取软链接信息并在新mfs里创建同名软链接</span></span><br><span class="line">            make_soft_link(current)</span><br><span class="line">        <span class="keyword">else</span>:</span><br><span class="line">            <span class="comment"># 在mfs下属文件夹中该目录不存在则创建</span></span><br><span class="line">            dst = current.replace(SOURCE_ROOT, TARGET_ROOT)</span><br><span class="line">            <span class="keyword">if</span> os.path.exists(dst) == <span class="literal">False</span>:</span><br><span class="line">                os.system(<span class="string">'mkdir -p '</span> + dst)</span><br><span class="line">            doing_copy_file(current)</span><br><span class="line">           </span><br><span class="line">    print(<span class="string">'done!'</span>)</span><br></pre></td></tr></table></figure>
        
            <div class="donate-container">
    <div class="donate-button">
        <button id="donate-button">赞赏</button>
    </div>
    <div class="donate-img-container hide" id="donate-img-container">
        <img id="donate-img" src data-src="/img/donate.jpg">
        <p> 感谢鼓励 </p>
    </div>
</div>
        
        <br>
        <div id="comment-container">
        </div>
        <div id="disqus_thread"></div>

        <div id="lv-container">
        </div>

    </div>
</div>

    </div>
</div>


<footer class="footer">
    <ul class="list-inline text-center">
        
        

        
        <li>
            <a target="_blank" href="http://weibo.com/2983694150">
                            <span class="fa-stack fa-lg">
                                  <i class="iconfont icon-weibo"></i>
                            </span>
            </a>
        </li>
        

        

        

        

    </ul>
    
    <p>
        <span>/</span>
        
        <span><a href="https://www.10000h.top">10000H</a></span>
        <span>/</span>
        
        <span><a href="https://niexiaotao.com">Xiaotao&#39;s Page</a></span>
        <span>/</span>
        
        <span><a href="#">It helps SEO</a></span>
        <span>/</span>
        
    </p>
    
    <p>
        <span id="busuanzi_container_site_pv">
            <span id="busuanzi_value_site_pv"></span>PV
        </span>
        <span id="busuanzi_container_site_uv">
            <span id="busuanzi_value_site_uv"></span>UV
        </span>
        Created By <a href="https://hexo.io/">Hexo</a>  Theme <a href="https://github.com/aircloud/hexo-theme-aircloud">AirCloud</a></p>
</footer>




</body>

<script>
    // We expose some of the variables needed by the front end
    window.hexo_search_path = "search.json"
    window.hexo_root = "/"
    window.isPost = true
</script>
<script src="https://cdn.bootcss.com/jquery/3.3.1/jquery.min.js"></script>
<script src="/js/index.js"></script>
<script async src="//busuanzi.ibruce.info/busuanzi/2.3/busuanzi.pure.mini.js"></script>


    <script>
        /**
         *  RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR PLATFORM OR CMS.
         *  LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables
        */
        if( '' || '')
        var disqus_config = function () {
            this.page.url = '';  // Replace PAGE_URL with your page's canonical URL variable
            this.page.identifier = ''; // Replace PAGE_IDENTIFIER with your page's unique identifier variable
        };

        (function() { // DON'T EDIT BELOW THIS LINE
            var d = document, s = d.createElement('script');
            s.src = 'https://airclouds-blog.disqus.com/embed.js';
            s.setAttribute('data-timestamp', +new Date());
            (d.head || d.body).appendChild(s);
        })();
    </script>



</html>
